OOCOHBB? B ESQ BE 

ED 096 862 BE 005 855 



AUTHOR 
TITLE 

TMSTITOTIOH 
PFPOPT NO 
PUB OAT? 
MOT? 



r Pilly, Richard R. 

Factors in Graduate School Performance. 
r duca*ionai Testing Service* Princeton, 

Feo 7a 
Sip, 



M.J. 



EfP* PFIC? 
DESCRIPTORS 



HF-fH.75 HC-S3. 15 PLUS POSTAGE 
'Academic Achievement; 'Achievement Bating; 
chemistry; English; 'Graduate Students; Graduate 
Study; 'Higher Education; Psychology; Research 
Projects: Student Characteristics; 'Student 
^valuation; Success Factors 



ABSTRACT 

critical incidents vere used to collect faculty 
ratings of graduate student performance in the departments of 
cheaistry, English, and psychology. Separate factor analyses in each 
of three fields produced eight factors that were fairly consistent 
vi»h respect to itea loadings across fields. Pactors were labeled 
independence and initiative, conscientiousness, critical facility, 
enthusiasm, research and experimentation, communication, teaching 
skills, and persistence. In addition to ratings of students, faculty 
vere asked to rate each incident with respect to its importance or 
relevance for success. Importance ratings were most similar for 
psychology and chemistry and least similar for chemistry and English. 
The implications for future studies of graduate student performance 
are discussed. (Author) 



RB-74-2 



FACTORS IN GRADUATE STUDENT PERFORMANCE 



Richard R. Reilly 



^'■ava?^"* 



PfRMISS-ON TO KMOOIlCE 
R,GMT t D MATSH.AI MAS BUN CHANTED B* 




TO IWC AND^WGANtfAtW OW»W 
UNDER AOB«MtNTS«.T» THt NATlONAljN 
ST.TUTE Of C0UCAT.ON 

Sj, RES PIHM.SSK3N O* T»£ CO"<*G»T 
OVW 



Tliis Bulletin is a draft for interoffice circulation 
Corrections ant * suggestions for revision are solicited. 
The Bulletin should not be cited as a reference without 
the specific permission of the author. It is automat i- 

i 

oil 1 y superseded upon formal publication of the material, j 



Educational Testing Service 
Princeton, New Jersey 
February 1974 



Factors in Graduate Student Performance 
Richard R. Rellly 

Abstract 

Critical Incidents were used to collect faculty ratings of 
graduate student performance In departments of chemistry, English, 
and psychology. Separate factor analyses In each of three fields 
produced eight factors which were fairly consistent with respect to 
item loadings across fields. Factors were labeled independence and 
initiative, conscientiousness, critical facility, enthusiasm, research 
and experimentation, communication, teaching skills and persistence. 
In addition to ratings of students faculty were asked to rate each 
Incident with respect to Its Importance or relevance for success. 
Importance ratings were most similar for psychology and chemistry and 
least similar for chemistry and English. The Implications for the 
findings for future studies of graduate student performance are discussed. 



Factors in Graduate Student Performance 

♦ 

Richard R# Reilly 

t>ie iriticartnctiieht technique; a inethod which ha* been used for 

many years in studying the performance of workers in industry* has been 

described by Flanagan (1954) as 

••• essentially a procedure for gathering certain important 
facts concerning behavior in defined situations. • • • the 
critical incident technique does not consist of a single - 
rigid set of rules governing such data collection* Rather 
it should be thought of as a flexible set of principles which 
must be modified and adopted to meet the specific situation 
at hand. 

Usually the procedure begins with the questioning of experts familiar 
with a specific job or task. The experts are asked to describe incidents 
from their experience which they consider indicative of unusually effective 
performance or unusually ineffective performance. The critical incidents 
ire collected and collated and may be put to a variety of uses* They may 
be categorized Judgmental ly and used to guide item writers in test development, 
for example. Critical incidents may also be used, as in the present study, 
as valuable tools in the development of criteria* 

The present study was the second phase of a project which began 
with the collection of a larfte number of critical Incidents of graduate 
student performance (Reilly, 1971). During this first phase a representative 
sample of graduate faculty from the fields of chemistry, English, and 
ps who logy were asked to provide several incidents or examples of behavior 
which caused them to raise or lower their estimate of the competence 
of i graduate student 11 (Reilly, 1971). The large number of critical 



incidents which resulted from this survey was reduced to a final list 

of *>i Incidents or examples of graduate student performance through 

the elimination of items which were inappropriate , top vague, or redundant* 

The present report describes an attempt to derive empirical factors or 

d imens tv ns~6T graduat e " student per f ormahce "by Examining the" factor structure 

of the critical incidents when used to describe the performance of actual 

graduate students. The underlying rationale for the method used in this 

study may be summarized in the following set of premises* 

!. An evaluation of the performance of a graduate srtudei* % lite 
the evaluation of any other performance, is usually Msed on 
a series of behaviors manifested by the students and known to 
the evaluator. Thus, an overall rating is an attempt by the 
evaluator to summarize this series of behaviors , very often 
with a single number. 
2* Only certain behaviors matter. These are the critical incidents 
which cause the evaluator to shift his judgment as to the 
competency of the student. Actually, most behaviors encountered 
by an evaluator are probably Irrelevant in that they do not 
cause any shift in judgment. 

3, Since opinions may vary as to what behaviors characterize an 
exceptional graduate student one reasonable approach Is to ask 
a representative sample of experts (i.e., faculty) to describe 
behaviors which have, in the past, caused them to raise or lower 
their estimate of \ graduate student's competence. 

4. Once a representative sample .of incidents has been collected 

they may be used to describe the performance of specific students. 
Hy determining which kinds of incidents tend to co-occur, specific 



males might be suggested for use as criteria. At the same 
tin*?, faculty members can provide some indication as to the 
Importance. of each Incident foi performance in their field. 

I'urj^ose and Method 

.nt-TnaJor objective of -the present investigation was to define, 

empirically* a set of criterion dimensions upon which graduate faculty 
base Judgments of student performance. A second goal was to examine 
differences among faculty with r aspect to how important they regard 
various incidents of student performance. Since the critical Incidents 
had been collected from the fields of chemistry, Eng!ish,and psychology 
it was decided to sample the same three fields for the second phase 
of the study. These three fields were originally chosen because they 
represented a broad range of the disciplines in which graduate degrees 
•are offered and, in addition, were three of the largest areas of study 
(t'artter, 196S). 

A faculty rating booklet (see Appendix) was prepared and sent 
to ail department chairmen for the three fields in member institutions 
of the Council of Graduate Schools (CGS). In addition, a letter was sent 
M the chairmen explaining the project and inviting their participation. 
The letter explained that participation meant asking their faculty 
members to cooperate by filling out the booklets and then returning the 
questionnaires to ETS. At the same time another letter was mailed to 
the Graduate Deans of all CGS institutions explaining the project and 
requesting their support. 

The faculty rating booklet was composed of four separate sections. 
In each of the first three sections the faculty member was asked to 
describe the performance of three specific graduate students chosen 
by the professor by indicating whether he observed a particular behavior 



<t>B), observed its opposite (NKc;) # or observed both the behavior and its 
opposite (if both boxes were blank it wis assumed that neither the behavior 
nor Its opposite was observed) • The raters were asked to rate a below 
average student in the first section, an average student in the second 
sec Hon , an d an a bove average student in the last _se_c t ion _to_ ensure _soae___ 
variation in the overall level of ratee performance. In the fourth 
section faculty were ask«»d to ?aie the importance or relevance of each 
incident for performance in tle!r field* 
Re sults 

l*ata were collected fron a total of 227 departments and 1299 faculty 
members across the three fields. Table 1 gives the number of depart* 
taettts and faculty members participating for each field* Since each 
faculty member rated J students the total number of students rated was 
389 7. For the first i sections each item was scored 1 if the box labeled 
oB was checked* -1 if the box labeled NEC was checked* and 0 if either 
no check was made or if both boxes were checked* The fourth section was 
scored on a 4 point scale ranging from 1 for an incident which had little 
or no importance to a 4 for an i rem which was considered extremely important* 

Insert Table I about here 

Ratings of Students 

Frequency distributions for each item were produced for students 
rated below average, average, and above average in each discipline* 
Tables 2, i, and 4 present the means and standard deviations for each 
item* Inspection of the tables is rather tedious but, in general, the 
item means are in logical order with negatively stated items having 
highest means for the below avenge student, while positively stated 
i te ms have highest means lor tlie above average student* Some of the 



Items having to do with research* particularly laboratory research % 
have small standard deviations within English departments and small 
mean differences between the above and below average category* 



Insert Tables 2 t 1 f and 4 about here 



A separate factor analysis of performance ratings was performed 
within each field* Initial computations were done using the method of 
principal components, and factors with eigenvalues exceeding 1*0 were 
retained for the rotational step* All rotations were done using Kaiser 9 s 
varimax method (Kaiser, 1958) which rotates the original factors 
orthogonally to a more interpretable solution. The method yielded 
H factors far both chemistry and psychology and 7 factors for the English 
departments. In general, there was a fair degree of similarity in the 
solutions obtained. Tables 5 through 12 present each of the eight 'factors, 
matched across fields with respect to the highest item loadings (see Appendix A 
for complete factor patterns)* The items included in the tables were the 
5 items loading highest on each factor within each discipline* Items with 
loadings below .40 were not Included* 



Insert Table 5 about here 



The first factor was quite similar across disciplines (see Table 5) 
and lias been called an Independence -inu Initiative factor since most of 
the items appear to reflect these two related trait** Since it emerged 
fir*t this factor u» counted for more of the variance than any remaining 
factor in all three disciplines* Thi. suggests that graduate students 
are probably wore heterogeneous In the.r independence and initiative 
than in any other aspect of performance* 



A "conscientiousness" factor emerging* second for data collected from 
chemistry departments appeared quite similar to factors emerging for 
Knglish departments and for psychology departments (see Table 6). 
Conscientiousness was the term chosen to describe this factor because many 
of the items reflect care and responsibility in professional work. It 
is worth noting that the item, "Became distracted by non-academic, non- 
professional interests," also loaded highly on this factor suggesting that 
the element of professional commitment may underlie the behaviors listed 
in Table 6. 

Insert Table 6 about here 

For want of a better term the name Critical Facility was given to 
the second factor emerping In English departments (see Table 7). Similar 
factors were found in both psychology and chemistry departments. Taken 
together, the items with high loadings in all three departments (i.e., 
Items 29, lo, 28, 2">, and ») seem to suggest an underlying personality 
trait. An individual scoring high on this factor, and thus consistently 
exhibiting this pattern of behaviors, would appear to be an Individual 
who holds strong opinions and is aggressive but lacks the ability or 
desire to separate what is relevant from what is irrelevant In his work. 
In addition, such an individual would perform rather poorly in the normal 
give- and- take criticism occurring In a graduate department. 

Insert Table 7 about here 

Involvement was the term given to the factor presented in Table 8* 
The item with the uniformly highest loading was "seldom if ever engages 
In informal contacts with faculty or fellow graduate students. 11 Th*s 
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underscores the importance of the informal socialization process in 
graduate edueat ion* 



I nser-t Table -H ^about here 



The Ph.D. is by definition a research degree and thua a factor 

loading on items related to research might be expected. Table 9 shows 
the items with highest loadings on the Research and Experimentation 
factor. As might be expected more of the items loaded highly in the 
pyschology and chemistry analysis than that for English* Curiously, 
however, Item* having to Jo with laboratory research had high loadings 
for factor V for the English departments* "Laboratory" nay have been 
interpreted differently by English faculty than by psychology or chemistry 
professors, but enough variance did occur in these items for it to be 
statistically related to the research factor. 



Insert Table 9 about here 



The ability to communicate both verbally and in writing is a factor 
of some importance in all three fields* A Communications factor emerged 
second in psychology and third in chemistry* As mentioned previously, 
only 7 factors emerged for the English departments but the second factor 
for English could be viewed as a combination of the Critical Facility 
factor discussed earlier and the Communications factor presented in 
Table 



Insert Table 10 about here 



The final two factors are presented in Tables 11 and 12* The 
Teach I ro* SMlls factor consists of only three items the first two of 



which i leurly reflect an Interest and ability in teaching. The last 
Item "devoted considerable time to helping other students with problems" 
had lower loadings but Is both logically and statistically related to 
the teaching skills factor. 

Insert Tables 11 and 12 about here 



Hie last factor consisted of only two similar items reflecting 
persistence. The only other item which might logically be related 
to this factor was item 40, "Did not hesitate to repeatedly ask questions 
of faculty until he fully understood an issue," which had higher loadings 
on the involvement factor discussed earlier. 
Importance Ratings 

The mean importance ratings of each incident for each discipline 
are shown In Table 13. The most similar disciplines in terms of mean 
Importance ratings were chemistry and psychology, while the least 
similar were chemistry and English. Table 14 presents the correlations 
between disciplines obtained when items were treated as observations 
and the mean importance ratings as scores. In general, the largest 
differences in Judged Importance occurred for those items related to 
empirical research, with such Items rated much higher by chemistry 
and psychology faculty than English faculty. One other item on which 
there was a large difference Is noteworthy. The item "submitted a 
paper which raerelv summarized what he had read" was judged to be a 
highly relevant behavior by English faculty who gave it a mean rating 
of J. 41 as contrasted with 2. J4 tor chemistry and 2.64 for psychology. 
A report which Is a mere summarisation is probably more acceptable in 
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department., ot ■ homlstry and to some extent, psychology, where empirical 
tovilts 't -..it u% mat 1 . 1 1 formulae are discussed, with extensive inter- 
pret hi -i im • *!t i «•• t »iff either not appropriate or not possible. 

Insert tables 1 I and 14 about here 

:v h-fi!.i. us »t "lmpt>rtanoe M might be distinguished for purposes 
tt.u -tad-., •.«. tvj.t? is the importance of various behaviors as 
judged hv experts, which was the definition used for Section IV of the 
Faculty Hating Booklet (see Appendix B).A second definition is related 
to the statistical properties of an item. An item becomes "Important" 
in defining a particular factor, for example, because it co-varies with 
other items. It should be recognized that these two different kinds of im- 
portance are not necessarily related. A good example would be a behavior 
which everyone agreed was highly Important but which, In practice, had 
little variation. The ability to read and speak English would probably 
be Judged Important for success in graduate study but would have little 
statistical Importance since there would be almost no variation. 

one way of relating the two cypes of importance is by looking at 

the Judged relevance of items loading highest on each of the factors. 

Accordingly, the mean importance ratings for the sets of items listed 

In Tables 5 through 12 were used to compute average item importances 

for each factor. Table 15 presents the results. 

Insert Table 15 about here 

Independence and Initiative which was the first factor to emerge 
In all three disciplines received almost Identical mean Item importances. 



-lo- 
in ill cases the mean rating was slightly above the "moderately important" 
category on the scale. As might he expected the Communications and 
Critical Facility factor are rated much higher In importance by English 
faculty than by either chemistry or psychology faculty. The reverse 
was true tor the Research and Experimentation factor which was moderately 
Important for psychology and chemistry but moderately unimportant for 
English. The only other factor which showed large differences in 
importance was the Teaching Skills factor which was judged less important 
by chemistry faculty than by English or psychology. 
Conclusions and Recommendations 

The results presently described suggested a number of potential 
criteria which might be used in studying graduate student performance, 
of the eight factors identified Initiative and independence accounted 
for the most variance in each of the three disciplines. The remaining 
factors were labeled conscientiousness* critical facility, involvement, 
research and experimentation, communication, teaching skills and 
persistence. Importance ratings of the items were most similar for 
chemistry and psychology and least similar for chemistry and English. 

The relatively consistent factors which emerged in each of the three 
disciplines studied suggest that the present results might be a first 
step In the development of a ~2t of rating scales which could be used 
as criteria of graduate student performance. Such scales could take 
sever j 1 forms but two approaches are suggested. The first would involve 
developing more critical incidents specifically for the factors Identified, 
particularly those with only 2 or J items with high loadings. The 
Incidents could then be used in a check list fashion to rate students, 
and scores could be derived separately for each of the factors. A 
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seeond approach would be to develop behavtorally anchored rating scales 
similar to those devised by Hilton, Kendall, and Spreclier (1970) for 
graduate study In business. Since factors have been Identified and 
named, some of the developmental steps outlined by Hilton, Kendall 
and Sprecher could presumably be eliminated. More incidents would have to 
be written particularly for the mid-points on the scales. Either 
approach could lead to a useful set of criterion measures for research 
In graduate schools. 

As a final cautionary note It Is important to recognise that the 
present data were based on behaviors or incidents that in practice 
distinguish the successful from the unsuccessful graduate student. 
Though It Is well-known (and often lamented by researchers) that graduate 
students are relatively homogeneous with respect to variables such as 
Intelligence, it is important to bear this in mind in interpreting the pres- 
end results. Aspects of performance highly correlated with the "explicit 
selectors" Omlliksen, 1950, p. 130) in graduate admissions might be 
perceived as normative or go unnoticed In graduate school where selection 
Is extreme, while in other less selective contexts these same aspects 
of performance might he critical in separating the more effective 
performers from the average or below par. 
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Table 1 

Total Numbers of Departments and Faculty Participating by Field 

Field Departments Faculty 
Chemistry 81 397 
English 64 443 
Psychology 82 459 
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TABLE 14 

Correlations Among Mean Department Ratings of Importance 

English Psychology 
Chemistry .27 .87 

English .42 
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TABLE 15 

Mean Item Ratings tor Items Loading Highest on 
Factors Emerging for All Three disciplines a 



Factor 


Chemistry 


English 


Psychology 


Independence and Initiative 


3. U 


3.14 


3.16 


Consc lent iousness 


2.76 


2.69 


2.97 


Critical Facility 


2.62 


3.22 


2.91 


Research and Experimentation 


3.01 


2.26 


3.00 


Involvement 


2.94 


3.09 


2.95 


Coramunicat ions 


2.74 


3.41 


3.01 


Teaching 


2.09 


2.51 


2.45 


Persistence 


3.02 


2.84 


3.05 


Overall Item Mean 


2.86 


2.94 


2.97 


All item means for factors are 


based on items 


listed in 


Tables 5 



through 12* The overall Item mean Is based on ratings for all 52 items* 
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FACULTY RATING BOOKLET 



The members of the Graduate Record Examinations Board, which 
formulates policies guiding the Graduate Record Examinations, have for 
some time been aware that research findings on the validity of our exam- 
inations are Inadequate, being few and not representative. We would like 
to see that more studies are accomplished, but in attempting to do so 
encounter problems arising trom both the scarcity of suitable data and 
Inadequate criteria of graduate performance. Hence we are conducting a 
study of complex regression systems, both Bayesian and least squares, 
which will allow us to pool the experience of many graduate departments 
In a way not feasible with existing methods. We have also developed plans 
for the very crucial criterion research which Is needed in order to best 
relate the Examinations to graduate school performance. We are therefore 
requesting your participation in an attempt to determine the importance 
and frequency of occurrence of various kinds of observations that might 
be made with respect to & student 9 s performance. Ve would like to use 
data reflecting your experience arid can do bo based on your responses to 
the Items in this booklet. The need for better research information in 
graduate admissions Is urgent and this study may in part help us to provide 
it. 

You will notice that this booklet is composed of four sections, each 
utilizing the same list of fifty-two observations. In each of the first 
three parts you are asked to use the list to describe the performance of 
a graduate student with whom you have had contact in the recent past. 
In choosing the students to describe we ask that you choose one student 
whom you considered to be above average, one average student, and one 
below average student. For each student check the box labeled OB If you 
made the observation described. If you made an observation opposite to 
that described, check the box labeled NEG (if both the observation and 
Its opposite occurred, you may check both of these categories). If you 
had no opportunity to make. the observation described or its opposite or 
If the item is not applicable to your field, leave the Item blank. 

In the final section of this booklet you are asked to rate each 
observation in terms of what ycu consider to be its Importance for your 
discipline. If you think that there are additional observations which 
are important for your field and should have been included on our list, 
you may describe these on the final page of the booklet. You may also 
use this page to give us any comments you may have regarding this study. 

Your efforts to provide us with data based on your experience are 
sincerely appreciated and we hope that the results of our program of study 
will be e>f some use to your Institution, and graduate education in general, 

In the r,«Lar future. 



Faculty Member Same 



Section I 



For this first section please use the list of observations below to 
describe the performance of a specific graduate student whom you consider 
t° »e below average relative to other graduate students you have encountered. 

Consider each observation in terms of your experience with the student 
being described. If you made an observation such as the one described, 
check the box labeled OB. If you made an observation opposite to that 
described, check the box marked NEC, and If both the observation and its 
opposite were made, check both boxes. If the observation Is not appli- 
cable to your field, or if you had no opportunity to make an observation 
such as the one described, leave the item blank. 



Student Name* 



Year of Graduate Study (check one) i 

*•* _ 3rd 5th or more 

2nd 4th Graduated 



OB NEC 

U U *• Rigidly followed a research plan when more flexibility 
would have been advantageous. 

n n 2. Tills student's willingness to pursue unasslgned readings 
was reflected by a broader than average knowledge of 
most topics. 

CD 3. Consistently offered well founded and constructive 
criticism of other students' presentations. 

n I I 4. When making a judgment or reaching conclusions, this 

student supported his position with carefully documented 
research. 

I"! n 5. Was able to master a difficult research technique in an 
unusually short period of time. 

QC] 6. During informal discussions with faculty this student 
displayed a genuine inte *st in and committment to his 
field. 

O CD 7* Was careless in reporting cata. 

("1 I 1 8. In conducting research thin student relied too heavily 
on one particular research tool. 



If for any reason you wish to preserve th" anonymity of the ratee, you may 
leave this blank. 
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OB NK(. 

Pi 1 j 9. Displayed an inability to write competently. 

, r "l ( 1 10. Failed on one or more occasions to complete a major 
assignment on time. 

O CJ 11. In writing a report this student synthesized material 
from two independent fields. 

□ Q 12. Shoved himself to be unfamiliar with a major research 

tool In his field. 

O O 13 • Was able t0 consider several markedly different approaches 
to a research problem and view them objectively before 
choosing one. 

f"l 14. Performed an experiment without making proper checks. 

| j j j 15. Handles even the most menial assignment (e.g., paper 
grading) with care and responsibility. 

□ D 16. Talks at great length In class but exhibits little 

understanding of material on papers and tests. 

ft n 17. Was unable to formulate a testable hypothesis from a 
theoretical analysis. 

0 D 18. Developed an original way of handling a research problem. 

[j □ 19. When presenting a paper' handled a difficult topic with 
considerable skill. 

f~"| n 20. Became more proficient in a useful outside field under 
his own initiative. 

fj f""j 21. Became distracted by non-academic, non-professional 
interests. 

[] [] 22. Was heavily dependent on direction from faculty and 

appeared unable to undertake any independent investigation. 

C"| ["""| 23. Devoted considerable time to helping other students 
with problems. 

("1 i"l 24. Exhibited carelessness with laboratory equipment. 

j ) | j 25. Although able to criticize studies with facility was 
unable to suggest better alternatives. 

| ; PI 26. Despite discouraging advice from faculty this student 
pursued his interest or ideas and was successful. 

1 i r~] 27. Seldom, if evtr en;;.»j«.es in informal contacts with faculty 
' or f ellow graduate btudeuts. 
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NEC 

□ 


28. 


Was often unable to consider new Ideas objectively because 
of strongly held prejudices. 


□ 


n 


29. 


Repeatedly made irrelevant remarks during class or seminar 
discussion. 


□ 


n 


30. 


This student usually did more than the required work. 


p— 1 

□ 


□ 


31. 


Submitted a report which was incomplete. 


□ 


□ 


32. 


Independently planned and executed a study which made a 
worthwhile contribution to his field. 


1— 1 


□ 


33. 


Was unable to effectively apply a research technique. 


□ 


□ 


34. 


This student displayed s fsmlllsrlty with the latest 
developments in his f isld. 


□ 


□ 


35. 


Showed imagination and originality in teaching a 
traditionally dull topic to an undergraduate class. 


1—1 

u 


□ 


36. 


Learned an important research skill on his own. 


□ 


□ 


37. 


When this student ssked s question it wss slways relevant 
and usually perceptive. 


□ 


□ 


38. 


Became quickly and enthusiastically involved in a resesrch 
project. 


□ 


□ 


39. 


Vss unwilling or unable to accept criticism. 


□ 


□ 


40. 


Did not hesitate to repeatedly ask questions of faculty 
until he fully understood sn issue. 


□ 


□ 


41. 


Submitted a paper or report which failed to address 
ths assigned Issues. 


□ 


□ 


42. 


Stimulated great Interest and enthusiasm in undergraduate 
courses in which he was an Instructor. 


□ 


□ 


43. 


Vss able to articulately defend his position and ideas. 


□ 


□ 


44. 


Avoided challenging courses or work. 


□ 


□ 


45. 


Conducted a data analysis which was inappropriate for the 
experiment as designed. 


□ 


n 

LJ 




Atr.Amnr.iMi ro earrv out noorlv olAnned research. 


□ 


□ 


47. 


despite one or more setbacks continued to work on research 
until it was successfully completed. 


□ 


□ 


48. 


Was unprepared for a seminar. 
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*9. Submitted a paper which merely summarized what he had 
read* 



OC] 50. Showed an ability to examine carefully an author's 
premises and frame of reference before accepting 
conclusions* 

n n 51. Asked for more work when none was assigned. 

□ □ 52. Presented ideas In a seminar, paper, or test in a poorly 
organised and disjointed fashion. 
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section II 



For this second section please use the list ss you did In Section I 
but this time for a specific student whoa you consider to be shout average 
relative to other students whom you have encountered. 



Student 



Year of Graduate Study (check one) : 

3rd 5th or more 

2w> 4th Graduated 



OB NBC 

Q Q 1* Rigidly followed a research plan when more flexibility 
would have been advantageous. 

n D 2. This student's willingness to pursue unasslgned readings 
was reflected by s brosder than average knowledge of 
most topics. 

f"l n 3. Consistently offered well founded snd constructive 
criticism of other students' presentations. 

n n 4. When making a judgment or reaching conclusions, this 

student supported his position with carefully documented 
research. 

("1 n 5. Was able to master a difficult research technique in an 
unusually short period of time. 

n fl 6. During Informal discussions with fsculty this student 
displayed s genuine Interest In and committment to his 
field. 

O D 7 * Was careless In reporting data. 

f~1 n 8. In conducting research this student relied too heavily 
on ons particular research tool. 



If for any reason you wish to preserve the anonymity of the ratee, you may 
leave this blank. 
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OB MF»; 

□ c *. 

□ D 10. 

□ □ u. 

□ □ 13. 

□ □ 16. 

□ □ 18. 

□ □ 1». 

□ □ 20. 

□ □ 21 - 

□ □ 22- 

□ □ 23. 

♦ 

□ □ 2*. 

□ G 25. 

C □ 26. 

□ H 27. 

ERIC 



Displayed an inability to write competently* 

Failed on one or more occasions to complete a major 
assignment on time. 

In writing a report this student synthesized material 
from two independent fields* 

Shoved himself to be unfamiliar with a major research 
tool in his field* 

Was able to consider several markedly different approaches 
to a research problem and view them objectively before 
choosing one* 

Performed an experiment without making proper checks* 

Handles even the most menial assignment (e*g*» paper 
grading) with care and responsibility* 

Talks at great length in class but exhibits little 
understanding of material on papers and tests* 

Was unable to formulate a testable hypothesis from a 
theoretical analysis* 

Developed an original way of handling a research problem* 

When presenting a paper handled a difficult topic with 
considerable skill* 

Became more proficient In a useful outside field under 

his own initiative* 

Became distracted by non-academic, non-professional 
interests* 

Was heavily dependent on direction fron faculty and 
appeared unable to undertake any independent investigation* 

Devoted considerable time to helping other students 
with problems* 

Exhibited carelessness with laboratory equipment* 

Although able to criticize studies with facility was 
unable to suggest better alternatives* 

Despite discouraging advice fron faculty this student 
pursued his interest or ideas and was successful. 

Seld<v. t if ever «n..i?es in informal contacts with faculty 
or fellow graduate tudents* 



-B8- 



□ 


□ 


28. 


Was often unable to consider new ideas objectively because 
of strongly held prejudices. 


□ 


□ 


29. 


Repeatedly made irrelevant remarks during class or seminar 
discussion. 


□ 


□ 


30. 


This student usually did more than the required work. 


□ 


□ 


31. 


Submitted * report which was imcomplete. 


□ 


□ 


32. 


Independently planned and executed a study which made a 
worthwhile contribution to his field. 


□ 


□ 


33. 


Was unable to effectively apply a research technique. 


□ 


□ 


34. 


This student displayed a familiarity with the latest 
developments In his field. 


□ 


□ 


35. 


Showed Imagination and originality in teaching a 
traditionally dull topic to an undergraduate class. 


□ 


□ 


36. 


Learned an important research skill on his own. 


□ 


□ 


37. 


When this student asked a question it was always relevant 
and usually perceptive. 


□ 


□ 


38. 


Became quickly and enthusiastically Involved In a research 
project. 


□ 


□ 


39. 


Was unwilling or unable to accept criticism. 


□ 


□ 


40. 


Did not hesitate to repeatedly ask questions of faculty 
until he fully understood an Issue. 


□ 


□ 


41. 


Submitted a paper or report which failed to address 
the assigned issues. 


□ 


□ 


42. 


Stimulated great Interest and enthusiasm In undergraduate 
courses in which he was an Instructor. 


□ 


□ 


43. 


Was able to articulately defend his position and Ideas. 


□ 


□ 


44. 


Avoided challenging courses or work. 


□ 


□ 


45. 


Conducted a data analysis which was inappropriate for the 
experiment as designed. 


n 

L-J 


n 


46. 


Attetnotdd to carrv out ooorlv olanned retearch. 


□ 


□ 


47. 


Despite one or more setbacks continued to work on research 
until it was successfully completed* 


□ 


□ 


48. 


Was unprepared for ■> seminar* 
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[ *1 f7] 49. Submitted a paper which merely summarized what he had 
read. 

PI Q 50* Showed an ability to examine carefully an authors 
premises and frame of reference before accepting 
conclusions* 

| | I I Sit Asked for more work when none was assigned* 

fl I j 52. Presented ideas in a seminar, paper, or teat in a poorly 
organized and disjointed fashion. 
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Sectlon III 



For this third section please use the list as you did in Section I 
but this time for a specific student whom you consider to be above average 
relative to other students whom you have encountered. 



Student Name* 



Year of Graduate Study (check one): 

3rd 5th or more 

2nd 4th Graduated 



0* MSG 

0 O 1* Rigidly followed a research plan when more flexibility 

would have been advantageous. 

n I 1 2. This student 9 a willingness to pursue unasslgned readings 
was reflected by a bro&der than average knowledge of 
most topics. 

1 1 I 1 3. Consistently offered well founded and constructive 

criticism of other students 9 presentations. 

n n 4. When making a judgment or reaching conclusions, this 

student supported his position with carefully documented 
research. 

I I | | 5. Was able to master a difficult research technique in an 
unusually short period of time. 

("""] n 6. During informal discussions with faculty this student 
displayed a genuine interest in and committment to his 
field. 

fn I ) 7. Was careless in reporting data. 

n | \ 8. In conducting research this student relied too heavily 
on one particular research tool. 



If for any reason > ju v^ish to preserve the anonymity of the ratec, you may 
leave this blank. 
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OB 

□ 


NfcG 

□ 


9. 


□ 


□ 


10. 


□ 


□ 


11. 


□ 


□ 


12. 


□ 


□ 


13. 


□ 


□ 


14. 


□ 


□ 


15. 


□ 


□ 


16. 


□ 


□ 


17. 


□ 


□ 


18. 


□ 


□ 

• 


19. 


□ 


□ 


20. 


□ 


□ 


21. 


□ 


□ 


22. 


□ 


□ 


23. 


□ 


□ 


24. 


□ 


i 


25. 




1 

> . 


26. 




4^ 


?7. 



9. Displayed an inability to write competently. 

Failed on one or more occasions to complete a major 
assignment on time. 

In writing a report this student synthesized material 
from two independent fields. 

Shoved himself to be unfamiliar with a major research 
tool in his field. 

Was able to consider several markedly different approaches 
to a research problem and view them objectively before 
choosing one. 

14. Performed an experiment without making proper checks. 

Handles even the most menial assignment (e.g., paper 
grading) with care and responsibility. 

Talks at great length in class but exhibits little 
understanding of material on papers and tests. 

Was unable to formulate a testable hypothesis from a 
theoretical analysis* 

18. Developed an original way of handling a research problem. 

When presenting a paper handled a difficult topic with 
considerable skill. 

Became more proficient in a useful outside field under 
his own Initiative. 

Became distracted by non-academic, non-professional 

interests. 

Was heavily dependent on direction from faculty and 
appeared unable to undertake any independent investigation, 

Devoted considerable tine to helping other students 
with problems. 

24. Exhibited carelessness with laboratory equipment. 

Although able to criticize studies with facility was 
unable to suggest better alternatives. 

Despite discouraging advice from faculty this student 
pursued hi» Interest or ideas and was successful. 

Seldom, if ever engages in informal contacts with faculty 
or fellow graduate students. 
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□ 


Shi. 

□ 


26, 


i i 






□ 


□ 


30. 


i i 


1 




□ 


□ 


32. 


□ 


□ 


33. 


□ 


□ 


34. 


□ 


□ 


35. 


□ 


□ 


36. 


□ 


□ 


37. 


□ 


□ 


38. 


□ 


□ 


39. 


□ 


□ 


40. 


□ 


□ 


41. 


i i 


n 
i i 




□ 


□ 


43. 


n 
i i 


n 


**H « 


□ 


□ 


45. 


□ 


□ 


46. 


□ 


□ 


47. 


□ 




48. 



of strongly held prejudices* 



discussion- 



worthwhile contribution to his field. 

33. Was unable to effectively apply a research technique, 

This student displayed a familiarity with the latest 
developments in his field* 

Shoved imagination and originality in teaching a 
traditional i> duii topic to an undergraduate class* 

36. Learned an important research skill on his own. 

When this student asked a question it was always relc 
and usually perceptive. 

Became quickly and enthusiastically involved in a re* 
project. 

39* Was unwilling or unable to accept criticism. 

Did not hesitate to repeatedly ask questions of facul 
until he fully understood an issue. 

Submitted a paper or report which failed to address 
the assigned issues. 

Stimulated great interest and enthusiasm in undergrad 
courses in whicn he was an instructor* 



experiment as designed. 
46. Attempted to carry out poorly planned research. 

Despite one or more setbacks continued to work < 
until it was successfully completed. 

48. Was unprepared for a seminar. 
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Q Q 49. Submitted a paper which merely summarized what he had 
read. 

□ □ 50. Showed an ability to examine carefully an author *s 

premises and frame of reference before accepting 
conclusions. 

□ □ 51. Asked for more work when none was assigned. 

D CI 52 • Presented ideas in a seminar, paper, or test In a poorly 
organized and disjointed fashion. 
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Section IV 

On this final listing of observations you are asked to consider each 
carefully in terms of its importance for your particular discipline and 
check the number beside it which you think is most appropriate, where: 

4 • an example of an instance which is extremely important for 
performance in your field; 

3 * an example of an instance which has moderate Importance for 
performance in your field; 

2 « an example of an instance which, though not completely irrelevant, 
carries little real importance for performance in your field; and 

1 • an example of an instance which has no Importance at all in your 
field and/or is totally irrelevant to it. 

On the last page of this booklet you can, if you wish, describe 
observations which you think are important for performance in your field, 
but which were not included in the Use* You may also feel free to make 
any comments you have concerning this study* 



4 3 2 1 

D D D D 1# Rigidly followed a research plan when more flexibility 

would have been advantageous* 

| I Q [] Q 2. This student's willingness to pursue unas signed readings 

was reflected by a broader than average knowledge of 
most topics* 

C71 D O O Consistently offered well founded and constructive 

criticism of other students 1 presentations* 

I 1 [][][] 4. When making a judgment or reaching conclusions, this 

student supported his position with carefully documented 
research* 

D D O C3 Was a ^le to master a difficult research technique in an 

unusually short period of time* 

pi PI 1 j r*| 6. During informal discussions with faculty this student 

displayed a genuine interest in and committment to his 
field* 

j j r~) Qj Q 7. Was careless in reporting data. 

n Q D [I] * # In conductin 8 research this student relied too heavily 

on one particular research tool* 
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4 3 2 1 

□ □ □ □ Displayed an inability to write competently. 

□ □□□ 

10. Palled on one or wore occasions to complete a major 
assignment on time. 

Q □ □ Q 11. In writing a report this student synthesized material 

from two Independent fields. 

O □ □ Q 12. Showed himself to he unfamiliar with a major research 

tool in his field. 

a □ □ □ 13. Was able to consider several markedly different approaches 

to a research problem and view them objectively before 
choosing one. 

□ □ □ □ 14. Performed an experiment without making proper checks. 

□ □ □ □ 

15. Handles even the most menial assignment (e.g., paper 
grading) with care and responsibility. 

D DDQ 16. Talks at great length In class but exhibits little 

understanding of material on papers and tests. 

□ □ □ □ 17. Was tumble to formulate a testable hypothesis from a 

theoretical analysis. 

D D D D 1* • Developed an original way of handling a research problem. 

□ □ □ □ 19. When presenting a paper handled a difficult topic with 

considerable skill. 

□ □ □ □ 

20. Became more proficient In a useful outside field under 
his own initiative. 

D D D D 21. Became distracted by non-academic, non-professional 

interests. 

□ □ □ □ 22. Was heavily dependent on direction from faculty and 

appeared unable to undertake any Independent Investigation. 

□ □ □ □ 23. Devoted considerable time to helping other students 

with problems. 

□ □ □ □ 24. Exhibited carelessness with laboratory equipment. 

□ □ □ □ 25. Although able to criticize studies with facility was 

unable to suggest better alternatives. 

□ □ □ □ 

26. Despite discouraging advice from faculty this student 
pursued his interest or ideas and was successful. 

□ □ □ □ 27. Seldcn, if ever engages in informal contacts with faculty 

or fellow graduate students. 



4 

□ 


3 

□ 


2 
□ 


1 

□ 


28. 


m « — mm 

□ 


□ 


□ 


□ 


29. 


□ 


□ 


□ 


□ 


30. 


□ 


□ 


□ 


□ 


31. 


LJ 


n 


1 1 


n 

LJ 


JZ. 


LJ 


n 

LJ 


PI 
1 1 


n 
i i 




□ 


□ 


□ 


□ 


34. 


n 

L_J 


n 


n 

LJ 


n 

LJ 




n 


n 


n 

LJ 


n 

LJ 


JO. 


□ 


□ 


□ 


□ 


37. 


n 


n 


n 


n 


Jo. 


n 

LJ 


n 
i i 


n 
i — i 


n 

LJ 




□ 


□ 


□ 


□ 


40. 


□ 


□ 


□ 


□ 


41. 


□ 


□ 


□ 


□ 


42. 


□ 


□ 


□ 


□ 


43. 


□ 




□ 


□ 


44. 


n 

LJ 


n 
i — i 


n 

LJ 


n 




□ 


□ 


□ 


□ 


46. 


□ 


□ 


□ 


□ 


47. 


□ 


□ 


□ 


□ 


48. 
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Was often unable to consider new Ideas objectively because 
of strongly held prejudices. 

Repeatedly made irrelevant remarks during class or seminar 
discussion* 

This student usually did more than the required work. 

Submitted a report which was incomplete • 

Independently planned and executed a study which made a 
worthwhile contribution to his field. 

Was unable to effectively apply a research technique. 

This student displayed a familiarity with the latest 
developments in his field. 

Showed imagination and originality in teaching a 
traditionally dull topic to an undergraduate class. 

Learned an important research skill on his own* 

When this student asked a question It was always relevant 
and usually perceptive. 

Became quickly and enthusiastically Involved in a research 
project. 

Was unwilling or unable to accept criticism. 

Did not hesitate to repeatedly ask questions of faculty 
until he fully understood an issue. 

Submitted a paper or report which failed to address 
the assigned Issues. 

Stimulated great Interest and enthusiasm in undergraduate 
courses in which he was an instructor* 

Was able to articulately defend his position and Ideas. 

Avoided challenging courses or work. 

Conducted a data analysis which was inappropriate for the 
experiment as designed. 

Attempted to carry out poorly planned research. 

Despite one or more setbacks continued to work on research 
until it was successfully completed. 

Was unprepared for a seminar. 
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4 3 2 1 

49* Submitted a paper which merely summarized what he had 
read* 

□ □ □ □ 50* Showed an ability to examine carefully an author's 

premises and frame of reference before accepting 
conclusions* 

□ □ □ □ 51* Asked for more work when none was assigned* 

□ □ □ □ 52. Presented ideas in a seminar, paper, or test in a poorly 

organized and disjointed fashion. 
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